Method and a system for programming an industrial robot

ABSTRACT

A method for programming an industrial robot. The robot is taught a path having waypoints located on or in the vicinity of an object. An image of the object is obtained. Information is obtained about the position of a pointer pointing at points. The position of the points relative to the object is determined. A point being pointed out is stored as a waypoint. A graphical representation is generated of the stored waypoint and the point being pointed out. A view is displayed including the object and the graphical representation. A system includes a pointer for pointing out points on or in the vicinity of the object, a position determiner determines the position of the points relative to the object, a camera delivers an image of the object and the pointer, a graphical generator generates graphics, a display displays a view including the object and graphical representation, and an activator.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending PCT/IB03/01939 filed 20May 2003.

FIELD OF THE INVENTION

The present invention relates to a method in connection with programmingof an industrial robot, comprising teaching the robot a path which has anumber of waypoints located on or in the vicinity of an object to beprocessed by the robot. The processing of the object could be any typeof surface processing application carried out by an industrial robot,such as painting, polishing, grinding, and welding.

The present invention also relates to a computer program productdirectly loadable into the internal memory of a computer, comprisingsoftware code portions for performing the steps of the method accordingto the invention, when said product is run on a computer.

The present invention also relates to a computer readable medium havinga program recorded thereon, where the program is to make a computerperform the steps of the method according to the invention when saidprogram is run on the computer.

The present invention further relates to a system for use in connectionwith programming of an industrial robot, the system comprising apointing member adapted to point out points on or in the vicinity of theobject, a position determining means, determining the position of saidpoints in relation to the object, a camera adapted for delivering animage of the object and the pointing member, a graphical generator,generating graphics by means of a computer and a display member,displaying a view comprising the object and said graphicalrepresentation projected on the object.

PRIOR ART

Robots are often used for processing the surface of an object. Existingpractice for programming a robot involves teaching the robot a sequenceof waypoints. The waypoints define the path, which the robot shallfollow during the processing of the object. A waypoint comprisesthree-dimensional position and three-dimensional orientationinformation. The robot is taught how to perform the task by being guidedthrough the various waypoints along the desired operating path duringthe programming. These waypoints are stored as instructions into amemory in the robot control unit. During operation of the robot, theprogram instructions are executed, thereby making the robot operate asdesired.

Generally, the robot is programmed to perform a task by an humanoperator who manually jogs the robot to the desired positions along thepath using a movable programming unit, a so-called teach pendant. Anindustrial robot usually carries an end-effector comprising a tool forperforming the processing, for instance a welding tool or a paintingtool. During programming of the path, the operator has to position andorient the end-effector in each waypoint along the path. Alternatively,if a 3D CAD model of the object exists, a person with a computerscientist background teaches the waypoints in a robot simulation system,so called off-line programming. Existing methods based on the CADapproach provide visual feedback in a virtual world, i.e. arepresentation of the real world and the real object.

Either way, the teaching process is time consuming, troublesome, errorprone, and in almost all cases requires several iterations before theprogram is acceptable. The CAD approach is costly and not alwaysintuitive to use. The complexity of the CAD-system requires the operatorwho is programming the robot to have knowledge about computer science.Such a person usually has little or no knowledge about the process. Inthe case of manual teaching, the control of the process is oftendifficult to optimize, but it has the advantage of indirectly utilizingimplicit process knowledge of the operator. Another disadvantage withthe manual teaching method is that it does not include any visualfeedback to the operator, visualizing what has been programmed. Theoperator must use the trial and error method until the program isacceptable. For example, when the robot is to be taught how to paint anobject and the entire surface of the object must be covered with colour,it is impossible for the operator, without running the programafterwards, to see if he has missed some part of the surface. Thus, anumber of iterations are required before the quality of the processing,in this example the painting, is satisfactory.

From the Japanese patent JP10011122 an improved method for teaching anindustrial robot is known. The method includes presenting a visualfeedback to the operator of the response resulting from an operationbefore the robot is operated. The visual feedback is representedtogether with the real object. The method comprises: measuring thecurrent state of the robot and its environment by a CCD camera, theoperator inputs robot operation information by the teaching pendant,estimating a response after the operation of the robot based on theinputted information, converting the estimated operation intoinformation to be displayed as a picture, preparing picture informationrelated to the response of the robot based on the information related tothe camera and the estimated information, synthesizing the estimatedresponse based on the picture information measured by the CCD camera andthe picture information related to the response of the robot, anddisplaying a view comprising the robot, its environment and theestimated response of the robot. Thus, this method shows the nextoperation to be performed by the robot, but it does not provide anyvisual feedback to the operator regarding what he has been programmed.

SUMMARY OF THE INVENTION

The object of the present invention is to provide an improved industrialrobot programming solution, which reduces the teaching time for theteaching process and increases the quality of the processing result inthe first iteration.

According to one aspect of the invention, the object is achieved bymeans of the initially defined method, comprising: obtaining an image ofthe object, obtaining information about the position of a pointingmember pointing at points on or in the vicinity of the object,determining the position of the points in relation to the object basedupon said obtained information, storing the point being presentlypointed out by the pointing member as a waypoint upon receiving arecording signal, generating a graphical representation of the storedwaypoint and the point being presently pointed at by the pointingmember, and displaying a view comprising the object and said graphicalrepresentation projected on the object based on the image of the objectand the generated graphical representation of the points. Preferably,the image of the object is obtained by means of a camera.

This method improves the teaching process by visualizing recordedinformation about the waypoints of the robot path in real time and inrelation to the real object to be processed. Computer generatedinformation, representing the robot path taught by the operator, isdisplayed or visualized projected on the object. Thus, the operator getsreal-time visual feedback on his actions. The operator does not need anyparticular computer knowledge and he uses his knowledge and experienceabout the process during teaching of the robot. The operator can walkaround the object pointing out the waypoints of the path and immediatelysee the result of what he has been teaching. The method according to theinvention is more efficient than the prior art methods and supports theuse of implicit knowledge of the operator. It is no longer necessary torun the robot program to view the result of the programming. Thereby,the number of iterations needed and thus the teaching time is reduced.

According to an embodiment of the invention the method further comprisesgenerating a graphical representation of the path between the lateststored waypoint and the point presently being pointed out. Byvisualizing the path between the latest waypoint and the point, whichthe operator is presently pointing at, it is easier for the operator toestimate whether the point is suitable as a waypoint. If the operatorfinds the point suitable he records it, otherwise he moves the pointingmember to a new point. Preferably, the method further comprisesgenerating a graphical representation of the path between at least someof the stored waypoints. By showing the path between the waypoints, itis simple for the operator to see whether he has missed some part orparts of the object and to see which parts of the object are left tocover.

According to a further embodiment of the invention, the method furthercomprises: obtaining information about whether the processing shall beon or off between the waypoints, and generating a graphicalrepresentation of the path between the stored waypoints depending onwhether the processing shall be on or off. When processing is of thepath is either not visualized at all or visualized in a different wayfrom when processing is on. In certain applications, the paths betweensome of the waypoints shall not be processed and the paths between theother waypoints shall be processed. For example, during painting of anobject, the painting is on between some waypoints and the painting isoff between other waypoints. It is advantageous for the operator to beable to distinguish between the paths for which he has orderedprocessing on and the paths for which he has ordered processing off.

According to a further embodiment of the invention, the method furthercomprises: receiving information related to the processing, generating agraphical representation of the path between the stored waypoints, whichdepends on the received information related to the processing. Theinformation received is for example information about toolconfiguration. Thus, it is possible for the operator to estimate if thetool configuration chosen achieves the desired result. For an example,the information related to the processing can be which brush to be usedfor painting and the width of the path visualized between the waypointsdepends on the width of the chosen brush.

According to a further embodiment of the invention, the displaying stepfurther comprises: registering the generated graphical representation ofthe point to the image of the object to provide a composite augmentedreality image and displaying a view showing the object and the pointbased on the composite augmented reality image. An augmented realitysystem merges computer-generated graphics of objects with the user'sspace in the real world. In this embodiment, the computer generatedgraphics of the waypoints and paths are merged with the image of thereal object. Alternatively, the computer-generated graphics of waypointsand paths is displayed on see-through glasses worn by the operator. Thegenerated graphics are projected on the glasses so that the operator cansee the waypoints and paths projected in relation to the real world andthe object.

According to a further embodiment of the invention, the step determiningthe position of the point comprises: recognizing a first marker withinthe image, said first marker being positioned in a fixed relation to theobject, recognizing a second marker within the image, said second markerbeing positioned in a fixed relation to a pointing member pointing outthe point, determining the position of the point in relation to theobject based upon the first and the second marker. This method fordetermining the position of the point being pointed out by the pointingmember does not require any additional measuring equipment for measuringthe position of the point besides the two markers.

According to a further embodiment of the invention, the method furthercomprises: obtaining information about position and orientation of thepointing member and generating a graphical representation of theorientation of the pointing member in the point based upon the obtainedinformation. During programming of the robot, the operator also needs toteach the robot how to orientate the tool. The programming is furthersimplified if the orientations of the tool in previously programmedwaypoints are visualized to operator.

According to a further embodiment of the invention, the method furthercomprises generating robot programming code based upon the determinedposition of the waypoint. Robot program code is automatically generatedbased on the position of a recorded waypoint. Preferably, the robotprogramming code is also generated based upon the orientation of thepointing member in the point. Thus, the operator does not need anyknowledge about robot code generation to be able to teach the robot howto process the object. The operator teaches the robot how to perform theprocessing by pointing at the object and recoding the desired waypoints.

According to a further aspect of the invention, the object is achievedby a computer program product directly loadable into the internal memoryof a computer, comprising software code portions for performing thesteps of the method according to the invention, when said product is runon a computer. The computer program product is provided either on acomputer readable medium or through a network such as the Internet.

According to another aspect of the invention, the object is achieved bya computer readable medium having a program recorded thereon, where theprogram is to make a computer perform the steps of the method accordingto the invention, when said program is run on the computer.

According to still another aspect of the invention, the object isachieved by the initially defined system characterized in that itcomprises an activating member, storing a point as a waypoint uponactivating and that the graphical generator is adapted for generating agraphical representation of the stored waypoints and the point beingpresently pointed out by the pointing member based on the image of theobject and the positions of the points.

According to a further embodiment of the invention, the system comprisesmeans for indicating options for the configuration of the process andthe graphical generator is adapted for generating a graphicalrepresentation of the path between the stored waypoints based on theindicated option or options.

According to a further embodiment of the invention, the system comprisesa second activating member, which upon activating deletes a savedwaypoint. In one embodiment, the last waypoint is deleted. By such anactivating member, the operator may delete a wrongly saved waypoint andcorrect it by saving a new waypoint while he is programming the robot.

According to a preferred embodiment of the invention, the pointingmember is arranged separated from the robot and is adapted for beingheld and handled directly by an operator.

According to a further embodiment of the invention, the activatingmembers are arranged on the pointing member. This arrangement makes iteasy for the operator to point out the waypoints, record the waypoints,and delete them without releasing the grip of the pointing member.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be explained more closely by the description ofdifferent embodiments thereof and with reference to the appendedfigures.

FIG. 1 shows a system for use in connection with programming of anindustrial robot, according to an embodiment of the invention.

FIG. 2 a shows a front view of an example of a pointing member.

FIG. 2 b shows side view of the pointing member shown I FIG. 2 a.

FIG. 2 c shows top view of the pointing member shown I FIG. 2 a.

FIGS. 3 a-d show examples of augmented reality views showing a graphicalrepresentation of waypoints projected on a real object.

FIG. 4 shows the relation between the pointing member and two markers.

FIG. 5 shows flow chart of a method according to the invention forteaching an industrial robot a path.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

FIG. 1 illustrates a system for use in connection with programming of anindustrial robot according to the present invention. The systemcomprises a pointing member 1 utilized by a human operator 2 forteaching the robot how to process an object 3. The pointing member caneither be the processing tool itself, or preferably a deviceparticularly adapted for the purpose of teaching the processing of theobject. The pointing member could, for example, be a pen shaped deviceor an imitation of the processing tool, such as a paint-gun. Theteaching comprises teaching the robot a path including a number ofwaypoints given in a specific sequence and located on or in the closevicinity of the object 3. Each waypoint comprises the position of apoint on or in a close vicinity of the object, the orientation of aprocessing tool, such as a paintbrush, in the point, and data related tothe processing.

The operator holds the pointing member 1 in his hand and points at thepoint he wants to include in the path and orientates the pointing memberas he wishes the processing tool to be oriented in the point. Theoperator records a point by activation of an activation member, whichgenerates a recording signal. Upon receiving the recording signal thesystem stores the point as a waypoint. The pointing member 1 is providedwith a marker 5, called a pointing marker, attached thereto. Thepointing marker 5 is used for determining the position and orientationof the pointing member 1. Alternatively, the pointing device maygenerate positions and orientations in a number of different ways. Forexample, the pointing member can be a mechanical digitizer arm providingthe position and orientation of its tip or the pointing member mayinclude a sensor for determining position and orientation. Examples ofsensors and technologies for determining positions and orientations areaccelerometers, ultrasound, laser technology, and a Global PositioningSystem (GPS).

The system is based on image recognition and requires at least onereference marker 6. The function of the reference marker 6 is to make upa reference point for image recognition algorithms, and a referencepoint to which computer generated graphics should be related. The systemfurther comprises a camera 8. The camera 8 must be arranged so that, theobject 3 to be processed and the markers 6, 5 are in the field of visionof the camera. The camera can either be fixed in the space or movedaround. For instance the camera can be attached to the operator's heador elsewhere on his body. All types of cameras can be used, but thedemand for accuracy of position and orientation determines the type ofcamera. For example, a web camera, a video camera, or a CCD camera couldbe used. In another embodiment, more than one camera can be used, forexample two cameras could be used arranged so as to provide stereoscopicimages.

The camera 8 provides either an analogue or a digital video signal,which is transferred to a computer 10 including a processor. Theprocessor of the computer 10 is utilized for running the imagerecognition algorithms for determining the position and orientation ofthe pointing member 1 in relation to the object, for running algorithmsgenerating a graphical representation of waypoints pointed out by theoperator, and for combining the generated graphics of the waypoints withthe image provided by the camera 8 to provide a composite augmentedreality image.

A display member 12 visualizes the view of the camera combined with thecomputer-generated graphics, which, in this case, are the recordedwaypoints and the path between the points. In this embodiment, thedisplay member is a computer screen. Other types of displays, such as aPersonal Digital Assistant (PDA) screen and wearable glasses, could alsobe used. In the case of a PDA, the operator will hold the PDA in hishand while programming and the outcome of the programming will bevisualized on the PDA screen. In the case of wearable glasses, theoperator will wear the glasses on which the computer-generatedinformation is projected in relation to his position in the room. Inanother embodiment, the operator may have a head-mounted unit includinga head-mounted display with a camera attached to the display.

In the following the pointing member 1 will be described in moredetails. FIGS. 2 a-2 c show an example of an embodiment of the pointingmember 1 comprising an elongated body 14, one of its ends having a tip 4adapted for pointing out points on or in the vicinity of the object andthe other end comprising a handle 15 adapted for being gripped and heldby the operator. The pointing member 1 comprises a number of interactionmeans 17, 18, 20, 22 adapted for interaction between the operator andthe system. The interaction means can for example comprise push-buttons.One of the interaction means comprises an activating member 17, whichupon activation generates a recording signal for recording the point asa waypoint. In this embodiment the activating member 17 is a push-buttonarranged in the tip 4 of the body 14.

A second interaction means is used for indicating whether the processingshould be on or off between the waypoints. For example, if the processis painting, the interaction means indicates whether the paint is on oroff, or if the process is polishing the interaction means indicateswhether the polishing on or off. In this embodiment the interactionmeans is a push-button 18 arranged on the pointing member 1. For examplethe push-button 18 is of a type which is held while processing is on andreleased while processing is off. Alternatively, the push-button shiftsthe status from on to off, and vice versa.

The system is provided with a third interaction means for input ofinformation related to the process to the system. The third interactionmeans may be adapted for showing options for configuration of theprocess to be selected by the operator. In this embodiment the thirdinteraction means 20 is located on the upper side of the pointing member1 and is implemented as a scroll mechanism with a small display showingthe configuration of the process. For example, in a paint application,it is possible to select one or several options for the paintbrushstored in the robot controller. For other applications, for examplepolishing, the options relates to the configuration of the polishingbrush. Alternatively, the third interaction means can be implemented asa push-button where a new push selects the subsequent configuraton andhence, the operator must scroll through the entire list to get to thestart.

The fourth interaction means allows the operator to delete recordedwaypoints. The fourth interaction means comprises an activating member22, which upon activation deletes a recorded waypoint. In thisembodiment, the activating member 22 is a push-button arranged on thepointing member and allows the operator to delete the last recordedpoint. Hence, the operator can also delete a sequence of storedwaypoints starting with the last recorded waypoint. Alternatively, theinteraction means 17, 18, 20, 22 can be implemented as a part of thecomputer interface, so that the operator interacts both via the pointingmember for recording position and orientation and, via the computer 10.

The output from the interaction means is either transferred via a wireor wirelessly from the pointing member 1 to the computer 10. In anotherembodiment it is possible to implement the interaction means in thesoftware interface, for example as a group of radio buttons.

While teaching the robot, the operator records waypoints to be passedthrough during the processing of the object and he also records processrelated information in connection with the waypoints by using theinteraction means. During the teaching, the operator moves the pointingmember 1 and positions its tip at points on or in the vicinity of theobject, and if the operator is satisfied with the location of the point,he records it by activating the activating member 17. In relation toeach recorded waypoint, the operator uses the push button 18 to indicatewhether the processing should be on or off. The robot programmingutilizes the interaction means 20 to indicate the status of processrelated information, such as specification for tool configuration. Theinteraction means 18, 20 must be selected before the operator records anew waypoint. At the moment of recording, the status of the interactionmeans 18, 20 is read into the system. Alternatively, the status of theinput means 18, 20 are read after recording a new point.

As the operator records waypoints, representing the path on the object,computer generated information is visualized on the display 12. Thus,the operator gets real-time visual feedback on his actions. Before thefirst point is recorded, the display shows the real world without anycomputer generated information. When the operator records a point, thewaypoint is visualized, for example as a dot on the display. FIG. 3 a isan example of a view shown on the display during the teaching. The viewcomprises the object 3 and a graphical representation of the recordedwaypoints 26, 27 and the path 28 between the waypoints, projected on theobject. The displayed waypoints 26, 27 can be color-coded fordistinguishing between different points defined by specific sequences ofactions.

A line 30 between the last recorded waypoint 27 and the point 31 beingpresently pointed out by the pointing member, in this embodiment the tip17 of the pointing member, is visualized and displayed. This line 30 isvirtual, i.e. it is generated by the computer and visualized as computergenerated information and operates like a rubber band between the lastrecorded waypoint and the tip of the pointing member. The operator cansee the points and lines on the display 12. When the operator records anew waypoint, the virtual line 30 becomes fixed between the twowaypoints. The waypoints represent the positions of the robot on itspath for processing the object. In addition to the position, theorientation of the pointing member is also recorded and the recordedorientation represents the orientation of the robot tool. Hence, therecorded sequence of waypoints creates the basis for the robot path inthe robot program.

FIG. 3 b shows an embodiment of the invention, in which the width of thepath 32 is shown between the waypoints. The width of the path depends oninformation related to the processing, such as the chosen toolconfiguration, input by the operator via the interaction means. Forexample, if a narrow brush is selected the width of the visualized pathis narrow and if a wider brush is selected the width of the visualizedpath is wider. To display the width of the path makes it easier for theoperator to decide where to locate the next waypoint to avoid processingparts of the object, which has already been processed, and to noticeparts of the object, which should be processed. The operator canimmediately see if he has missed to cover some part or parts of theobject.

FIG. 3 c is a view illustrating a path comprising parts 33 for whichprocessing is on and other parts 34 for which processing is off. If theoperator indicates that the processing shall be off between twowaypoints, no path is shown between the waypoints and if the operatorindicates that the processing shall be on between two waypoints, thepath between the waypoints is visualized. Alternatively, indicativegraphics, such as a line or a colour-coded path, is shown between thewaypoints if the processing shall be off.

FIG. 3 d shows a view including a graphical representation 36 of therecorded orientation in the waypoints. The orientation is illustratedfor example by three orthogonal axes x, y, z.

By pointing with the tip of the pointing member the operator shows whereon the object to capture data, such as position and orientation andchanges in the status of the processing, and he also shows the correctsequence of waypoints and process related information. While theoperator makes a new robot program, he can delete and edit recordedpoints. The operator can delete the last recorded point while he isprogramming. After the programming is finished, the operator uses thepointing member to grab a recorded point by locating the pointing membernext to the point to be edited and activates the activating member 17.He then moves the pointing member with the waypoint to the new position.The rest of the graphics is updated simultaneously in real time. Theoperator releases the activating member 17 at the new position and thenew orientation.

To be able to generate a complete robot program, the system needsinformation about the configuration of the robot in the robot cell. Aconfiguration files must be available. The operator can, for instance,generate these data from the computer. The configuration file containsdata, such as configuration data, stored in the robot controller, adefinition of the robot tool, a definition of the object coordinatesystem, and other relevant data. Alternatively, the configuration filecan already exist and therefore be reduced. The operator should have thepossibility to edit an existing configuration file.

As the operator has finished the teaching sequence of waypoints and aconfiguration file exists, the computer takes the recorded waypoints andthe configuration file as input and generates a robot program with theright syntax for the specific robot controller. Then, a robot programgenerator generates the robot program. The robot program generator canbe any robot programming generator known to those skilled in the art.The tasks of the robot program generator are; locating the recordedpoints at the right places in relation to the real path of the robot,generating the right syntax, and translating the input to the robot to aprogramming language. The robot program is either downloaded to therobot controller so that the robot can execute the program or integratedinto a larger robot program as a module, in case the robot programcontains a sequence of different objects.

This embodiment of the system described herein is based on imagerecognition and utilizes two markers, the reference marker 6, which isprovided in a fixed relation to the object 3 and the pointing marker 5being attached to the pointing member 1. The markers 5, 6 must bedistinct and asymmetric so that they can be recognized from all angles.The pointing marker 5 must be provided with a pattern different from thepattern of the reference marker 6. The function of the reference marker6 is to provide the image recognition algorithm with a reference pointin the world to which the computer generated graphics should be related.One reference marker is enough to represent the reference coordinatesystem, but it is also possible to use more than one reference markerforming a pattern of reference markers, which defines the referencecoordinate system. If the camera is not fixed, for example worn by theoperator or otherwise moved around the object, several reference markersmay be necessary to enabling recognition of the reference coordinatesystem from various viewpoints. The image recognition algorithm requiresthat the reference marker or markers is/are within the view of thecamera.

Since the position and orientation of the pointing member, in thisembodiment, is determined by image recognition the system must comprisethe pointing marker 5. If the position and orientation of the pointingmember are determined by the other methods mentioned above, the systemonly needs the reference marker. The function of the marker 5 is torecognize where the pointing member is located in relation to thereference coordinate system given by the reference marker 6. The imagerecognition algorithms use the pointing marker 5 to generate positionand orientation of the pointing member based on the size and orientationof the pointing marker 5 in relation to the reference marker 6. Ifposition and orientation are determined by another method, the pointingmarker 5 is not necessary. For improving the accuracy of thedetermination of the position and orientation, two different methods forgenerating the position and orientation may be used as a complement toeach other.

The pattern on the markers 5, 6 must be asymmetric in order to determinethe orientation of the marker. The shape of the marker is given by theimage recognition algorithms and must be learned in advance. Thealgorithms cannot recognize a completely new and unknown marker. Themarkers need to be distinct, so that the image recognition algorithmsare able to distinguish between their positions in the world. Otherwise,the image recognition algorithms would not be able to determine which ofthe markers would represent the reference coordinate system and whichone would give the position and orientation in relation to the referencecoordinate system.

FIG. 4 illustrates the coordinate systems of the pointing marker 5 andthe coordinate system of the reference marker 6. It is to be noted thatthe figure only illustrates one reference marker but the system may justas well include more reference markers or a pattern of referencemarkers. FIG. 4 also illustrates the coordinate system 40 of the camera8 in relation to the coordinate systems of the markers 5, 6. Therelationship between the marker 5 and the tip 4 of the pointing member 1must be known, so that the position and orientation of the tip can bedetermined. The image recognition algorithms recognize the pointingmarker 5. The position of the tip of the pointing member and theorientation of the body 14 of the pointing member in the referencemarker coordinate system is determined by the recognition of the markers5, 6. The recognition is based on the size and angle of the pattern onthe marker in relation to the known pattern learned by the imagerecognition algorithms. The position is given in 3D as [x, y, z]. Theorientation is given in 3D, for example as a rotation matrix [a 3×3matrix].

In case a sensor is used to determine the position and orientation, thepointing member provided with such a sensor need to be calibrated to thereference marker coordinate system in order to give the position andorientation in the correct coordinate system. If the pointing member isprovided with a sensor, which is combined with a marker attached to theinput device, a calibration is still necessary.

FIG. 5 is a flow chart illustration of the method and the computerprogram product according to an embodiment of the present invention. Itwill be understood that each block of the flow chart can be implementedby computer program instructions. In block 60 a continuous stream ofvideo signals is received from the camera 8. Since the object 3 and themakers 5, 6 are in the view of the camera, the image obtained comprisesthe object and the markers. In block 62 an image is captured for use bythe image recognition. Alternatively, several images could be capturedfor improving the accuracy of the position and orientation. The markers5, 6 are recognized in block 64 by the image recognition and theposition and orientation of the pointing member are calculated inrelation to the location of the object or the reference markers, block66.

A graphical representation is generated of the point being presentlypointed out by the pointing member, in this embodiment, of the tip ofthe pointing member, block 68. Before the first waypoint is recorded, nopreviously generated graphic exists. In that case the graphic will onlyinclude the point being presently pointed out by the pointing member. Ifone or more waypoints have been previously recorded, graphics alreadyexist including the recorded waypoints, and the generated graphic of thepoint being presently pointed out are added to the existing graphics,block 70, 72. Graphic is generated representing a line between the lastrecorded waypoint and the point being presently pointed out.

As seen in block 74, the received video signal is combined with thegenerated graphics including registering the generated graphicalrepresentation to the image of the object to provide a compositeaugmented reality image. A view of the combined video signal andgraphics is displayed, block 76. If a recording signal is received, thestatuses of the interaction means are captured and the position,orientation and the captured status for the point being presentlypointed out are saved as a new waypoint, block 78, 80.

As the point is recorded as a waypoint, new graphics are generated basedon the captured status, block 72. The captured status is, for example,whether processing is on or off and information related to the process.The generated graphics comprise stored waypoints and the point beingpresently pointed out by the pointing member, in this embodiment, thetip of the pointing member. The display visualizes the view of thecamera combined with the generated graphics which in this embodiment arethe recorded waypoints, the path between the waypoints, the status ofthe process, i.e. the processing is on or off, and the width of thepath. The width of the path visualizes for example the paint stroke whenpaint is on.

The software used for implementing the method according to the inventionis partly based on software known to those skilled in the art. Forexample, the position and orientation may be generated in ARTtoolKitbased on the position and orientation of the pointing marker in relationto the reference marker. The ARTtoolKit is developed by WashingtonUniversity and the University of Hiroshima and is an open-sourcesoftware library that enables augmented reality applications to be builtusing accurate computer vision-based tracking techniques. For theapplication interface, the software Open GL may be used. OpenGL providesa library of 2D and 3D functions including modeling alterations, color,light and shade functions. Microsoft Vision SDK is a library for writingprograms to perform image manipulation and analyses on computers. Theaugmented reality software includes algorithms for drawing graphics,such as points and lines, transferring positions and orientationsbetween different coordinate systems, extracting and generating asequence list of positions and orientations, capturing process relatedinformation, and drawing advanced graphics, such as color-coded pointsand lines representing paint strokes with different widths andorientations.

The method according to invention the is an off-line programming method,i.e. the method may be used outside the robot and the robot controller

The present invention is not limited to the embodiments disclosed butmay be varied and modified within the scope of the following claims. Forexample, the present system is not limited to augmented reality systemsutilizing a composed video image but may also be utilized in see-throughaugmented reality systems, in which only the computer generated graphicsare presented to the user who views the graphics on a see-through lenswith the real world in the background. In a see-through system, thecameras would generate images, which could be utilized for locating themarkers within the field of view of the user. The only differencebetween the two systems is the translation of the camera coordinatesinto eye coordinates to compensate for the change in perspective betweenthe camera images and the actual real-world perspective of the user. Ineither case, computer generated graphics are registered to objects inthe real world. In the see-through case, the computer-generated graphicis combined with the real-world object by the user rather than in thecreation of a composed video image.

It is also possible to utilize the robot itself for moving the pointingmember, which for example, is a robot tool. The tool is then providedwith a pointing marker attached thereto. The operator moves the tool, orthe pointing member held by the robot, by means of the teach pendant.

1. A method in connection with programming of an industrial robot,comprising teaching the robot a path having a number of waypointslocated on or in the vicinity of an object to be processed by the robot,the method comprising: obtaining an image of the object, obtaininginformation about the position of a pointing member pointing at pointson or in the vicinity of the object, determining the position of thepoints in relation to the object based upon said obtained information,storing the point being presently pointed out by the pointing member asa waypoint upon receiving a recording signal, generating a graphicalrepresentation of the stored waypoint and the point being presentlypointed at by the pointing member, and displaying a view comprising theobject and said graphical representation projected on the object basedon the image of the object and the generated graphical representation ofthe points.
 2. The method according to claim 1, further comprising:generating a graphical representation of the path between the lateststored waypoint and the point presently being pointed out.
 3. The methodaccording to claim 1, further comprising: generating a graphicalrepresentation of the path between at least some of the storedwaypoints.
 4. The method according to claim 3, further comprising:obtaining information about whether the processing shall be on or offbetween the waypoints, and generating a graphical representation of thepath between the stored waypoints depending on whether the processingshall be on or off.
 5. The method according to claim 3, furthercomprising: receiving information related to the processing, andgenerating a graphical representation of the path between the storedwaypoints depending on the received information related to theprocessing.
 6. The method according to claim 1, wherein said displayingstep further comprises: registering the generated graphicalrepresentation of the point to the image of the object to provide acomposite augmented reality image and displaying a view showing theobject and the point based on the composite augmented reality image. 7.The method according to claim 1, wherein determining the position of thepoint comprises: recognizing a first marker within the image, said firstmarker being positioned in a fixed relation to the object, recognizing asecond marker within the image, said second marker being positioned in afixed relation to a pointing member pointing out the point, anddetermining the position of the point in relation to the object basedupon the first and the second marker.
 8. The method according to claim1, further comprising: obtaining information about position andorientation of the pointing member and generating a graphicalrepresentation of the orientation of the pointing member in the pointbased upon the obtained information.
 9. The method according to claim 1,further comprising: generating robot programming code based upon thedetermined position of the waypoint.
 10. The method according to claim9, further comprising: obtaining information about the orientation ofthe pointing member, determining the orientation of the pointing memberin the point based upon the obtained information and the robotprogramming code is generated based upon the determined orientation ofthe waypoint.
 11. The method according to claim 1, wherein the image ofthe object is obtained by means of a camera.